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Abstract 

We compute the channel capacity of non-binary fingerprinting under the Marking Assumption, in the 
limit of large coalition size c. The solution for the binary case was found by Huang and Moulin. They 
showed that asymptotically, the capacity is l/(c 2 21n2), the interleaving attack is optimal and the arcsine 
distribution is the optimal bias distribution. 

In this paper we prove that the asymptotic capacity for general alphabet size q is (q — l)/(c 2 2 In q). Our 
proof technique does not reveal the optimal attack or bias distribution. The fact that the capacity is an 
increasing function of q shows that there is a real gain in going to non-binary alphabets. 



1 Introduction 

1.1 Collusion resistant watermarking 

Watermarking provides a means for tracing the origin and distribution of digital data. Before 
distribution of digital content, the content is modified by applying an imperceptible watermark 
(WM), embedded using a watermarking algorithm. Once an unauthorized copy of the content is 
found, it is possible to trace those users who participated in its creation. This process is known 
as 'forensic watermarking'. Reliable tracing requires resilience against attacks that aim to remove 
the WM. Collusion attacks, where several users cooperate, are a particular threat: differences 
between their versions of the content tell them where the WM is located. Coding theory has 
produced a number of collusion-resistant codes. The resulting system has two layers: The coding 
layer determines which message to embed and protects against collusion attacks. The underlying 
watermarking layer hides symbols of the code in segmental of the content. The interface between 
the layers is usually specified in terms of the Marking Assumption, which states that the colluders 
are able to perform modifications only in those segments where they received different WMs. 
These segments are called detectable positions. 

Many collusion resistant codes have been proposed in the literature. Most notable is the Tardos 
code [13] . which achieves the asymptotically optimal proportionality m oc c 2 , with m the code 
length. Tardos introduced a two-step stochastic procedure for generating binary codewords: (i) For 
each segment a bias is randomly drawn from some distribution F. (ii) For each user independently, 
a or 1 is randomly drawn for each segment using the bias for that segment. This construction 
was generalized to larger alphabets in [14] . 



1.2 Related work: channel capacity 

In the original Tardos scheme |13j and many later improvements and generalisations (e.g. [161 
Q31 [31 OH M 13 HS1 H3)> users are found to be innocent or guilty via an 'accusation sum', a 
sum of weighted per-segment contributions, computed for each user separately. The discussion 
of achievable performance was greatly helped by the onset of an information-theoretic treatment 

The 'segments' are denned in a very broad sense. They may be coefficients in any representation of the content 
(codec). 
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of anti-collusion codes. The whole class of bias-based codes can be treated as a maximin game 
between the watermarker and the colluders [2J |5J [TJ, independently played for each segment, 
where the payoff function is the mutual information between the symbols x\, . . . , x c handed to the 
colluders and the symbol y produced by them. In each segment (i.e. for each bias) the colluders 
try to minimize the payoff function using an attack strategy that depends on the (frequencies of 
the) received symbols x\, . . . ,x c . The watermarker tries to maximize the average payoff over the 
segments by setting the bias distribution F, 

It was conjectured [7J that the binary capacity is asymptotically given by l/(c 2 21n2). The con- 
jecture was proved in [TJ[8]. Amiri and Tardos pQ developed an accusation scheme (for the binary 
case) where candidate coalitions get a score related to the mutual information between their 
symbols and y. This scheme achieves capacity but is computationally very expensive. Huang 
and Moulin [B] proved for the large-c limit (in the binary case) that the interleaving attack and 
Tardos's arcsine distribution are optimal. 

1.3 Contributions and outline 

We prove for alphabet size q that the asymptotic fingerprinting capacity is ggj^ - • Our proof makes 
use of the fact that the value of the maximin game can be found by considering the minimax game 
instead (i.e. in the reverse order) . This proof does not reveal the asymptotically optimal collusion 
strategy and bias distribution of the maximin game. 

In Section [2] we introduce notation, discuss the information-theoretic payoff game and present 
lemmas that will be used later. In Section [3] we analyze the properties of the payoff function in 
the large-c limit. We solve the minimax game in Section U In Section [S] we discuss the benefits of 
larger alphabets. 

2 Preliminaries 

2.1 Notation 

We use capital letters to represent random variables, and lowercase letters to their realizations. 
Vectors are denoted in boldface and the components of a vector x are written as Xi. The expec- 
tation over a random variable X is denoted as Ex ■ The mutual information between X and Y is 
denoted by I(X; Y), and the mutual information conditioned on a third variable Z by I(X; Y\Z). 
The base-q logarithm is written as log g and the natural logarithm as In. If p and <x are two 
vectors of length n then by p CT we denote Jl™=i Pi' ■ If c is a positive integer and er is a vector of 
length n of nonnegative integers with sum equal to c then f°) denotes the multinomial coefficient 
i a ^{ - r - The standard Euclidean norm of a vector x is denoted by ||cc||. The Kronecker delta of 
two variables a and j3 is denoted by 5 a p. A sum over all possible outcomes of a random variable 
X is denoted by J^x- ^ n order not to clutter up the notation we will often omit the set to which 
x belongs when it is clear from the context. 

2.2 Fingerprinting with per-segment symbol biases 

Tardos [13) introduced the first fingerprinting scheme that achieves optimality in the sense of 
having the asymptotic behavior m cx c 2 . He introduced a two-step stochastic procedure for 
generating the codeword matrix X. Here we show the generalization to non-binary alphabets 
[14] . A Tardos code of length m for a number of users n over the alphabet Q of size q is a set 
of n length-m sequences of symbols from Q arranged in an n x m matrix X. The codeword 
for a user i £ {1, . . . , n} is the i-th row in X. The symbols in each column j € {1, . . . , m} are 
generated in the following way. First an auxiliary bias vector € [0, l] q with ^2 Pa = 1 

a 

is generated independently for each column j, from a distribution F. (The P^ are sometimes 
referred to as 'time sharing' variables.) The result is used to generate each entry of 
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column j independently: P [Xij = a] = . The code generation has independence of all columns 
and rows. 

2.3 The collusion attack 

Let the random variable SJ, G {0, 1, . . . , c} denote the number of colluders who receive the symbol 
a in segment j. It holds that ^ a <Ta = c for all j. From now on we will drop the segment index 
j, since all segments are independent. For given p, the vector S is multinomial-distributed, 

A CT | P = Prob[S = a\P = p] = (^Jp*- (1) 

The colluders' goal is to produce a symbol Y that does not incriminate them. It has been shown 
that it is sufficient to consider a probabilistic per-segment (column) attack which does not distin- 
guish between the different colluders. Such an attack then only depends on X, and the strategy 
can be completely described by a set of probabilities d y \ a £ [0, 1], which are defined as: 



y y\cr 



= Prob[F = y | £ = <r]. (2) 



For all er, conservation of probability gives J2 y ®y\<r = 1- Due to the Marking Assumption, a a = 
implies # Q | CT = and a a = c implies # a | CT = 1. The so called interleaving attack is defined as 
Q a \<r — CT Q /c. 



2.4 Collusion channel and fingerprinting capacity 

The attack can be interpreted as a noisy channel with input X and output Y . A capacity for 
this channel can then be defined, which gives an upper bound on the achievable code rate of a 
reliable fingerprinting scheme. The first step of the code generation, drawing the biases p, is not 
considered to be a part of the channel. The fingerprinting capacity C c (q) for a coalition of size c 
and alphabet size q is equal to the optimal value of the following two-player game: 

C c (q) = maxmin -I(Y; S | P) = maxmin - / F(p)I(Y; S | P = p)d q p. (3) 

F c F c J 

Here the information is measured in g-ary symbols. Our aim is to compute the fingerprinting 
capacity C c {q) in the limit (n —¥ oo, c —¥ oo). 

2.5 Alternative mutual information game 

The payoff function of the game ^ is the mutual information I(Y; S | P). It is convex in 9 (see 
e.g. [S]) and linear in F. This allows us to apply Sion's minimax theorem (Lemma [IJ, yielding 

maxmin I(Y; S | P) = minmax/(r; S | P) (4) 

F OF 

= mmmax/(y; S | P = p) (5) 

where the last equality follows from the fact that the maximization over F in ([JJ results in a 
delta distribution located at the maximum of the payoff function. The game ([3]) is what happens 
in reality, but by solving the alternative game ((5|) we will obtain the asymptotic fingerprinting 
capacity. 

2.6 Useful Lemmas 

The following lemmas will prove useful for our analysis of the asymptotic fingerprinting game. 



Lemma 1 (Sion's minimax theorem [12]): Let X be a compact convex subset of a linear topological 
space and y a convex subset of a linear topological space. Let / : X x y — > M be a function with 
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• f(x, •) upper semicontinuous and quasiconcave on y, Vx G X 

• fi'tV) lower semicontinuous and quasi-convex on X, Vy G y 
then mm X £ X max 9e3 j /(x, y) = max yey min xeX f(x, y). 

Lemma 2: Let M be a real n x n matrix. Then M T M is a symmetric matrix with nonnegative 
eigenvalues. Being symmetric, M T M has mutually orthogonal eigenvectors. Furthermore, for any 
two eigenvectors t>i 1 1>2 of M T M we have Mv\ _L Mv2- 

Proof: M T M is symmetric because we have (M T M) T — M T (M T ) T = M T M. For an eigenvector 
v of M T M, corresponding to eigenvalue A, the expression v T M T Mv can on the one hand be 
evaluated to v T Xv = X\\v\\ 2 , and on the other hand to ||Mv|| 2 > 0. This proves that A > 0. 
Finally, any symmetric matrix has an orthogonal eigensystem. For two different eigenvectors Vi, 
v 2 of M T M, with Vi _L v 2 , the expression vj M T Mv 2 can on the one hand be evaluated to 
vjA 2 v 2 = 0, and on the other hand to (Mvi) T (Mv 2 ). This proves Mv\ _L Mv 2 . □ 

Lemma 3: Let V be a set that is homeomorphic to a (higher-dimenional) ball. Let dV be the 
boundary of V. Let / : V — > V be a differentiable function such that dV is surjectively mapped to 
dV. Then / is surjective. 

Proof sketch: A differentiable function that surjectively maps the edge dV to itself can deform 
existing holes in V but cannot create new holes. Since V does not contain any holes, neither does 



Lemma 4 (Arithmetic Mean - Geometric Mean (AM-GM) inequality): For any n G N and any list 
x%, a?2) • • • j %n of nonnegative real numbers it holds that ^ Yn=i x i — \/ x i x 2 ■ ■ ■ x n . 

3 Analysis of the asymptotic fingerprinting game 

3.1 Continuum limit of the attack strategy 

As in [6] we assume that the attack strategy satisfies the following condition in the limit c — > oo. 
There exists a set of bounded and twice differentiable functions g y : [0, l] q — > [0,1], with y G Q, 
such that 

1. g a (cr/c) = for all a, cr 

2. x a = implies g a (x) = 

3 - Eo^^ 1 implies J2 a 9a(x) = 1. 

3.2 Mutual information 

We introduce the notation r y | p = Prob[F = y\P = p) = J2a-^y\o-^a-\ P = E S |p =p [^y|s]- The 
mutual information can then be expressed as: 



where we take the b&se-q logarithm because we measure information in g-ary symbols. Using the 
continuum assumption on the strategy we can write 



/(V). 



□ 
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3.3 Taylor approximation and the asymptotic fingerprinting game 

For large c, the multinomial-distributed variable X tends towards its mean cp with shrinking 
relative variance. Therefore we do a Taylor expansion^ of g around the point — = p: 

4f )- *« + S 2 ^'- - *) + sfB'- - - <*»>|Sg + ■ ■ ■ < 8 > 

We introduce the notation K for the (scaled) covariance matrix of the multinomial-distributed S, 

K a p = ^Cov (E a , S^) = 8 a pp a - p a pp- (9) 

For r y | p we then get 



Ty\p ~ E S |p 



*(f)]-*W + 55'-^^(^)- 

The term containing the 1st derivative disappears because E S | p [X — cp] = 0. The 0(l/cy/c) 
comes from the fact that (X — cp) n with n > 2 yields a result of order c™/ 2 when the expectation 
over X is taken. Now we have all the ingredients to do an expansion of I(Y; X | P = p) in terms 
of powers of i. The details are given in Appendix |XJ 

I(Y; X | P = p) = + f (11) 



2c In q \ c^/c 



Note that T(p) can be related to Fisher Information]! The asymptotic fingerprinting game for 
c — » oo can now be stated as 

= maxmin / F(p)T(p)d q p. (13) 

2c z m g F g J 

3.4 Change of variables 

Substitution of if © into JUJ gives 

Now we make a change of variables p a = u 2 and <7 a (p) = 7q(*0, w i tn w a € [0, 1], 7«(m) € [0, 1]. 
The hyperplane J2 a p a = 1 becomes the hypersphere J2 a u 2 = 1. For u on the hypersphere we 
must have J2 a 1a( u ) = 1- Due to the Marking Assumption, u a = implies 7 Q (w) = 0. The 
change of variables induces the probability distribution $(u) on the variable it, 

*(«)4F(p(«))JJ(2«a). (15) 

a 

In terms of the new variables we have a much simplified expression, 

T(«)=^{||V 7 ,|| 2 -(wV 7 ,) 2 }. (16) 

y 

where V stands for the gradient d/du. 



2 Some care must be taken in using partial derivatives d/dpp of g. The use of g as a continuum limit of 
is introduced on the hyperplane ~^2 a .Pa = 1, but writing down a derivative forces us to define g(p) outside the 
hyperplane as well. We have a lot of freedom to do so, which we will exploit in Section 13.51 

3 We can write T(p) = Tr[K(p)X(p)] , with I the Fisher information of Y conditioned on the p vector, T a p(p) = 

'dlngytp) \ f aingy(p) \ 



E9y(p) 1 

y 



dp ft 
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3.5 Choosing 7 outside the hypersphere 

The function g(p) was introduced on the hypersphere ^2 a Pa — 1, but taking derivatives d/dp a 
forces us to define g elsewhere too. In the new variables this means we have to define "y(u) not 
only on the hypersphere 'surface' \\u\\ = 1 but also just outside of this surface. Any choice will 
do, as long as it is sufficiently smooth. A very useful choice is to make 7 independent of \\u\\, 
i.e. dependent only on the 'angular' coordinates in the surface. Then we have the nice property 
u ■ \7"f y = for all y € Q, so that ([TO]) simplifies to 

T(«) = ]T||V 7q || 2 (17) 

a 

and the asymptotic fingerprinting game to 

CJq) = — - — maxmin f <S>(u)T(u)d q u. (18) 
2c z In q * 7 J 

3.6 Huang and Moulin's next step 

At this point 6 proceeds by applying the Cauchy-Schwartz inequality in a very clever way. In 
our notation this gives 

maxmin / §{u)T(u)d q u > max - — ^ min[ / JT(u)d q u] 2 , (19) 

* i J ' * ./ i^ dqu 1 J 



with equality when T is proportional to l/$ 2 . For the binary alphabet (q = 2), the integral 
/ yjT(u)d q u becomes a known constant independent of the strategy 7. That makes the min- 
imization over 7 to disappear: The equality in (|19p can then be achieved and the entire game 
can be solved, yielding the arcsine bias distribution and interleaving attack as the optimum. For 
q > 3, however, the integral becomes dependent on the strategy 7, and the steps of [6] cannot be 
applied. 

4 Asymptotic solution of the alternative game 

Our aim is to solve the alternative game to (|18|) , see Section 12.51 



C M = 7TT"i minmaxT(w). (20) 

2c z mq ~f u 

First we prove a lower bound on max„ T(u) for any strategy 7. Then we show the existence of 
a strategy which attains this lower bound. The first part of the proof is stated in the following 
theorem. 

Theorem 1: For any strategy 7 satisfying the Marking Assumption (u a = =>■ j a (u) = 0) and 
conservation of probability (||w|| = 1 =>■ 1 1 -y (ix) 1 1 = 1) the following inequality holds: 

max T(u) >q-l. (21) 

u: -u>0, I u\\ — 1 

Proof: We start with the definition of the Jacobian matrix J(u): 

J a p{u) = —5 ■ (22) 

aup 

In this way we can write: 



T(u) = Tr(J T J). 



(23) 
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The matrix J has rank at most q — 1, because of our choice u ■ S7~/ y = which can be rewritten as 
Ju = 0. That implies that the rank of J T J is also at most q — 1. Let Ai(tt), A2(it), . . . , A g _i(w) 
be the nonzero eigenvalues of J T J. Then 

9-1 

T(u) (24) 
i=i 

Let •Ui,'W2, • • • >v q -i be the unit-length eigenvectors of J T J and let dit^), du( 2 ), . . ., dti( 9 _i) be 
infinitesimal displacements in the directions of these eigenvectors, i.e. dun-s oc Vi. According to 
Lemma [2] the eigenvectors are mutually orthogonal. Thus we can write the (q — l)-dimensional 
'surface' element dS u of the hypersphere in terms of these displacements: 

9-1 

<ls » = n uncoil- ( 2s ) 

i=l 

Any change du results in a change d-y = Jdu. Hence we have d^fu\ = Jdu^y By Lemma [21 the 
displacements d7( 1 \,d7( 2 ) ) • ■ • T^n^q-i) are mutually orthogonal and we can express the (q — 1)- 
dimensional 'surface' element dS-y as 

9-1 9-1 



dS T = ni|d7 W ll =11 Vll Jdw wll 2 ( 2 6) 
i=l 1=1 

9-1 9-1 _ 

= n ^u T (l) J T Jdu (€} = n nd« (i) nv^ (27) 

i=l i=l 
9-1 

- d5 u Yl 7a~ (28) 
i=i 

We define the spatial average over u as Av u [/(«)] = J /(«) d5 u / J" d^. We then have 

where the inequality follows from Lemma [3] applied to the mapping "f(u). (The hypersphere 
orthant ||u|| = 1, u > is closed and contains no holes; the 7 was defined as being twice 
differentiable; the edge of the hypersphere orthant is given by the pieces where Ui = for some 
i; these pieces are mapped to themselves due to the Marking Assumption. The edges of the 
edges are obtained by setting further components of u to zero, etc. Each of these sub-edges is 
also mapped to itself due to the Marking Assumption. In the one-dimensional sub-sub-edge we 
apply the intermediate value theorem, which proves surjectivity. From there we recursively apply 
Lemma [3] to increasing dimensions, finally reaching dimension q— 1). 
Since the spatial average is greater than or equal to 1 there must exist a point where 
^/Ai(m*)A2(m*) . . . Aq_i(«*) > 1 . Now we apply Lemma 01 

9-1 

T(m») = A *K) > (9 - 1) q -yX 1 (u,)X 2 (u,) . . . A g _i(u*) > q - 1. (30) 

i=l 

The last inequality holds since y/x > 1 implies q ~y/x > 1. Finally max u T(u) > T(u*) > q— 1. □ 
Next we show the existence of a strategy which attains this lower bound. 

Theorem 2: Let the interleaving attack 7 be extended beyond the hypersphere ||u|| = 1 as j y (u) = 
■piy, satisfying u ■ V7,, = for all y. For the interleaving attack we then have T(u) = q — 1 for all 
u> 0, Hull = 1. 
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Proof: 

dj y {u) _ S ya UyU a 

(32) 
(33) 

where we used the property 8y a = S ya . For ||tx|| = 1 it follows that T(u) = q — 1. □ 
These two theorems together give the solution of the min-max game (f2"Uf . The main result of this 
paper is stated in the following theorem: 

Theorem 3: The asymptotic fingerprinting capacity C^°(q) in the limit c — > oo for an alphabet of 
size q is given by 

<*•<«> = m 

Proof: For any strategy 7, Theorem [T] shows that max u T(tt) > q — 1. As shown in Theorem [2j 
the interleaving attack has T(u) = q — 1 independent of u. Hence 

minmaxT(it) = q — 1 (35) 

is the solution of the min-max game. By Sion's theorem this is also the pay-off solution to the 
max-min game, as shown in Section l2~5l Substitution into (|20[) yields the final result. □ 
Remark: Any distribution function $(m) is an optimum of the minimax game, since T(u) is 
constant for the optimal choice of 7. 



5 Discussion 



We have proven that the asymptotic channel capacity is C^°(q) = c 2 9 2 ~in q • This is an increasing 
function of q; hence there is an advantage in choosing a large alphabet whenever the details of the 
watermarking system allow it. 

Some confusion may arise because of the difference between binary and g-ary symbols, and the 
'space' they occupy in content. Therefore we explicitly mention the following. The capacity is an 
upper bound on the achievable rate of (reliable) codes, where the rate measures which fraction of 
the occupied 'space' confers actual information. The higher the fraction, the better, independent 
of the nature of the symbols. Thus the rate (and channel capacity) provides a fair comparison 
between codes that have different q. 

The obvious next question is how to construct a <?-ary scheme that achieves capacity. We expect 
that a straightforward generalization of the Amiri-Tardos scheme Q] will do it. Constructions with 
more practical accusation algorithms, like |14j . do not achieve capacity but have already shown 
that non-binary codes achieve higher rates than their binary counterparts. 

When it comes to increasing q, one has to be cautious for various reasons. 

• The actually achievable value of q is determined by the watermark embedding technique 
and the attack mechanism at the signal processing level. Consider for instance a q = 8 code 
implemented in such a way that a 9-ary symbol is embedded in the form of three parts (bits) 
that can be attacked independently. Then the Marking Assumption will no longer hold in 
the q = 8 context, and the 'real' alphabet size is in fact 2. 
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• A large q can cause problems for accusation schemes that use an accusation sum as in [14] . 
As long as the probability distributions of the accusation sums are approximately Gaussian, 
the accusation works well. It was shown in [11] that increasing q causes the tails of the 
probability distribution to slowly become less Gaussian, which is bad for the code rate. On 
the other hand, the tails become more Gaussian with increasing c. This leads us to believe 
that for this type of accusation there is an optimal q as a function of c. 

The proof technique used in this paper does not reveal the asymptotically optimal bias distribution 
and attack strategy. This is left as a subject for future work. 
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A Taylor expansion of I{Y; £ | P = p) 

We compute the leading order term of I(Y; £ | P = p) from ([7]) with respect to powers of -. We 
write \og q g y = \ng y /\nq and, using ©, ]ng y (<r/c) = \n[g y (p) + e y ] = hxg y {p) +ln(l + e y /g y (p)), 
where we have introduced the shorthand notation 

a 1 \ - dg y (p) 1 ^ (Pgyip) 

£y= c^ ^T (Eq " CPa) + 2? ^ " CPa){ ^ ~ ^dp^ + ' • ■ (36) 

a a/3 r 

Higher derivative terms are omitted since they contain higher powers of 1 /c (even after the expec- 
tation over S is taken). Next we apply the Taylor expansion ln(l + x) = x —%- + ••■ , resulting 
in 

**<!>- ta *w + sfo- 45 + - (37) 

where we stop after the second order term since that is already of order - when we take the 
expectation over S. Using ([TU]) we get 



\nT ylp = \ng y (p) + ^^ + (38) 
9y (P) 




Now we combine all the ingredients, 

= Wp) + fB + ., ^__J_ + ...) (40) 

where in the first factor we stop at e y because when the expectation over S is applied, e y gives at 
least a factor of - and the terms in the second factor give at least a factor of -k= . 
Now E S |p =p [ey — Cy] =0 because E S |jj =p [S — cp] = and C, y was defined as the expectation 
over S of the second term in (|36|) . The expectation of the product E S |p =p [e y Cy] 1S of order ^ 

and so we drop it as well. The only remaining part of order i in (1401) is 2g j-j; and hence we end 
up with: 



A Taylor expansion of I(Y; S | P = p) 



11 



I(Y;-E\P = p) 

= in— E — TT E S|P= P [4] + f ^7= 
2\nq^-f g y [p) V C VC 



E 



8p a 

dg y {p) dg y (p) 



(41) 



(42) 



O 



1 



(43) 



where in the second step we expanded and took the square of only the first term in (|36p because 
the other combination of terms give rise to higher powers of -. 



